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TO  TRANSFORM  INTO  A  MORE  CAPABLE  INTELLIGENCE  COMMUNITY: 
A  PARADIGM  SHIFT  IN  THE  ANALYST  SELECTION  STRATEGY 


Summary 

Transnational  security  threats  such  as  terrorism  and  proliferation  of  weapons  of  mass 
destruction  bring  out  the  challenging  nature  of  intelligence  analysis.  We  need  people  who  can 
thrive  in  an  analytical  environment  characterized  by  uncertainty,  incompleteness  and  surprise. 
These  are  capabilities  more  innate  than  learned.  Yet  the  entry  process  into  the  all-source 
analysis  profession  does  not  screen  for  people  with  these  abilities.  We  need  to  improve  our 
analytical  work  force.  To  do  this,  we  need  a  systematic  abilities-based  pre-employment 
screening  program  for  intelligence  analysts. 

This  paper  provides  a  functional  justification  for  a  screening  program.  Such  a 
proposal  represents  a  paradigm  shift  from  the  conventional  way  of  recruitment  largely  based 
on  a  resume  and  an  interview.  The  form,  content,  costs  or  time  to  develop  an  actual  program 
is  beyond  the  scope  of  this  paper.  However,  an  implementation  plan  and  strategy  for 
measuring  its  effectiveness  are  proposed.  The  potential  benefits  of  such  a  screening  effort 
include  a  higher  concentration  of  analytical  focus  on  difficult  intelligence  problems,  a  greater 
chance  for  success  in  "connecting  the  dots,"  a  model  to  be  used  by  military  and  law 
enforcement  analytical  efforts,  and,  most  importantly,  a  smaller,  more  powerful  analytical 
work  force. 

Framing  the  Issue:  Capabilities-Based  Strategy 

Soon  after  the  terrorist  attacks  of  1 1  September  2001,  the  Defense  Department 
published  its  Quadrennial  Defense  Review  (QDR).  The  QDR  embodied  a  new  strategy 
intended  to  minimize  the  destructive  impact  of  strategic  surprise.1  The  main  thesis  of  the 
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strategy  is  to  shift  defense  planning  from  a  threat-based  to  a  capabilities-based  model. 
Expressed  differently,  the  intent  of  the  strategy  is  to  shift  planning  away  from  framing  conflict 
by  the  character  of  war  -  the  who,  where  and  why  -  to  framing  conflict  by  the  conduct  of  war 
-  the  how.  The  shift  in  strategy  is  an  attempt  to  reduce  the  risk  of  uncertainty  and  surprise. 

The  implications  for  intelligence  cannot  be  understated.  The  intelligence  community 
played  the  key  role  when  war  planning  was  framed  from  a  threat  perspective.  It  identified  the 
priorities  for  operational  and  strategic  planning  and  placed  responsibility  for  mission  success 
on  the  operational  forces.  Switching  to  a  capabilities  perspective  means  operational  forces 
play  the  key  role  in  identifying  resource  priorities  for  planning.  Consequently  a  much  greater 
responsibility  for  mission  success  now  lies  with  the  intelligence  community  who  has  to 
quickly  identify  the  who,  what  and  where.  With  a  threat-based  model,  intelligence  can 
operate  more  independently.  However,  in  a  capabilities-based  model,  it  must  operate 
“continuously”  and  in  “high-speed”  because  dependency  on  intelligence  is  required. 

Setting  the  Context:  Analytical  Challenges  of  Counterterrorism 

In  the  aftermath  of  the  terrorist  attacks  of  1 1  September  200 1 ,  Congress  asked  why  the 
intelligence  community  failed  to  warn  the  nation  of  this  attack.  The  post-mortem  of  1 1 
September  did  not  discover  any  new  problem  with  the  intelligence  analyst;  it  just  revealed 
existing  problems  in  a  very  public  way.4  Congress  concluded  analysts  were  not  able  to 
“connect  the  dots”  and  they  identified  a  number  of  organizational  and  resource  reasons  largely 
responsible.5  While  these  reasons  contributed  to  the  failure  in  understanding  the  threat 
environment,  the  thesis  of  this  paper  is  there  exits  a  more  fundamental  reason  for  failing  to 
“connect  the  dots.” 
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There  are  at  least  two  fundamental  reasons  for  failing  to  “connect  the  dots.”  First,  the 


classical  intelligence  challenge  is  “noise.”  By  “noise,”  we  mean  the  bulk  of  infonnation 
available  to  the  analyst  not  appearing  useful.6  As  we  shall  see  below,  knowing  what  is 
“useful”  is  not  necessarily  obvious  or  unambiguous.  Each  analyst  factors  his  or  her  own 
innate  biases  into  the  assessment  of  what  is  useful.  Additionally,  the  residual,  what  is  useful, 
may  be  “long  on  details  but  short  on  intent.”7  As  a  result,  the  analyst  must  therefore  interpret 
sparse  and  seemingly  unrelated  data  sets.  Second,  the  analyst  must  interpret  the  “noise”  to 
recognize  patterns,  a  task  whose  success  may  largely  be  influenced  by  the  innate  quality  of 
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intuition.  Improvements,  therefore,  need  to  capitalize  on  innate  qualities. 

Relying  on  the  Wrong  Model  to  Fix  the  Status  Quo 

Too  much  attention  has  been  paid  to  external  fixes  and  not  enough  to  internal  ones.  It 
is  the  internal  ones  that  are  the  most  important  to  fix.  President  George  W.  Bush  stated,  “Our 
security  will  require  the  best  intelligence. .  ,”9  However,  focusing  on  making  improvements  to 
the  “analyst”  is  especially  challenging  since  the  reasons  for  analytical  failures  often 
uncovered  in  post-mortems  are  caused  by  organizational  relationships,  structural  limitations 
or  technological  shortfalls.  In  light  of  these  findings,  public  policy  analysis  must  assume  the 
problems  exist  external  to  the  “analyst.”  Public  debate,  therefore,  has  missed  the  opportunity 
to  evaluate  this  “inner  sanctum”  of  the  intelligence  analysis  business,  which  is  why  it  has 
become  the  forgotten  transformation. 

Senator  Richard  Shelby,  as  Vice  Chairman,  Senate  Select  Committee  on  Intelligence, 
during  the  107th  Congress,  identified  organizational,  structural  and  technological  problems 
faced  by  the  analytical  community.  His  observations  were  made  in  the  Joint  Intelligence 
Committee’s  investigation  into  the  refonn  of  the  Intelligence  Community  following  the  1 1 
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September  terrorist  attacks.  He  cited  problems  in  infonnation-sharing,  intelligence-law 
enforcement  coordination,  counterintelligence-counterterrorism  coordination,  human 
intelligence,  and  technology  support  for  data  mining.1 

Since  virtually  every  aspect  of  the  analyst’s  external  environment  has  been  discussed, 
why  have  we  not  put  the  analyst’s  “internal”  environment  “under  the  microscope?”  The 
reason  is  relatively  straightforward.  Not  only  do  we  not  have  a  baseline  from  which  to 
compare  analyst’s  performance  but  we  also  do  not  have  an  agreed  upon  set  of  capabilities  by 
which  to  measure.  By  not  identifying  these,  there  are  a  number  of  consequences:  we  do  not 
really  understand  the  capability  set  of  whom  we  are  hiring  from  a  corporate  perspective;  we 
cannot  effectively  determine  if  an  analyst  is  doing  the  right  job  since  we  do  not  have  a 
consensus  expectation  of  capabilities;  and  we  then  are  quite  limited  in  making  use  of 
corrective  or  career  development  actions  against  analysts  since  objective  standards  are  not  in 
place.  A  desired  outcome  is  to  define  the  innate  capabilities. 

The  Community’s  Dilemma:  Defining  a  Capability  Set 

When  we  think  about  the  all-source  analyst,  it  is  usually  in  terms  of  things  they  do  and 
the  knowledge  they  possess.  Analysts  identify  a  problem,  find  infonnation  about  it,  think 
about  the  infonnation,  reach  a  conclusion  and  then  write  a  report  addressing  the  problem  for  a 
policy-maker  or  decision-maker.  And  there  are  training  and  educational  programs  that  target 
these  tasks  and  improve  their  knowledge  of  facts.  However,  if  we  think  about  the  analyst  in 
tenns  of  how  they  think  -  which  is  the  fundamental  “crown  jewels”  of  analytical  work  -  then 
we  immediately  must  acknowledge  its  importance  over  what  they  know. 

The  National  Security  Agency  has  made  some  progress  in  identifying  the  basic  traits 
needed  by  the  intelligence  analyst  and  separates  these  into  characteristics,  abilities,  skills,  and 
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knowledge.1 1  The  first  two  are  the  more  innate  capabilities  of  an  individual.  NSA’s 
characteristics  include  curiosity,  self-motivation,  excited  by  ideas,  a  keen  observer,  a  reader, 
capable  of  using  multiple  perspectives,  creativity,  playfulness,  a  sense  of  wonder,  can 
intensely  concentrate  and  questions  convention.  Abilities  include  pattern  recognition  and 
reasoning.  The  remaining  two  are  skills  learned  through  education  or  training. 

Academic,  business  and  other  researchers,  however,  have  applied  a  more  rigorous 
approach  to  identity  capabilities  for  intelligence  analysts.  Heuer,  a  psychological  researcher, 
states  we  are  not  conscious  of  unconscious  variables  -  like  perception,  memory  and 
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information  processing  -  which  introduce  biases  into  the  thinking  process.  The 
implications  for  Heuer’ s  premise  is  dramatic:  not  only  must  the  intelligence  analyst  deal  with 
uncertainty  in  the  external  world  but,  to  a  large  degree,  uncertainty  also  exists  within  the 
analyst’s  internal  world  -  the  thought  process.  Heuer  concludes  cognitive  biases  -  making 
errors  -  even  if  one  is  aware  of  them,  cannot  be  easily  overcome  nor  produce  a  more  accurate 
perception.  Heuer’s  observation  frames  the  community’s  dilemma:  we  hope  to  hire  the 
best  analyst,  but  have  no  effective  way  to  understand  their  deep-down  capability.  We  hope 
training  and  experience  will  compensate. 

How  the  analyst  perceives  the  world  is  a  very  important  consideration  in  detennining 
the  accuracy  of  understanding  the  world.14  Heuer  believes  the  most  important  perceptual  bias 
is  we  tend  to  perceive  what  we  expect  to  perceive.15  These  expectations  operate 
subconsciously  and  tell  the  analyst  what  to  look  for,  what  is  important  and  how  to  interpret 
what  they  see.  They  form  patterns  which  Heuer  calls  “mind-sets,”  that  predispose  the  analyst 
to  think  in  certain  ways.  Once  you  have  a  mind-set,  you  filter  the  world,  and,  more 
importantly,  mind-sets  become  self-reinforcing  and  become  difficult  to  break.  As  a 
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consequence,  new  infonnation  supports  old  ideas,  and  identifying  small  changes  leading  to 
new  conclusions  becomes  very  difficult.16 

Clarkson,  a  researcher  in  integration  of  intelligence  and  technology,  sought  to  identify 
obstacles  to  analytical  thinking  and  grouped  these  into  two  categories:  conceptual  and 
cognitive  obstacles.17  Clarkson’s  conceptual  obstacles  may  be  overcome  or  minimized 
through  education  and  training.  The  analyst  has  limited  infonnation  and  therefore  the 
representation  of  the  world  based  on  that  data  is  limited.  The  analyst  may  not  know  how  to 
interpret  information  for  which  there  is  no  historical  context.  The  analyst  may  not  be  able  to 
recognize  differences  between  data  if  they  are  similar.  Finally,  the  analyst  will  be  tempted  to 
impose  a  pattern  or  model  of  activity  for  which  many  variations  in  the  real  world  exist. 

Clarkson’s  cognitive  obstacles  are  more  innate.  Analysts  tend  to  work  with  narrow 
hypotheses  restricting  their  perspective.  The  downside  of  this  limitation  is  reliance  more  on 
hope  than  on  analytical  prowess.  A  related  obstacle  is  the  over-reliance  on  the  use  of  a 
hypothesis  to  explain  data.  It  prevents  the  analyst  from  seeing  a  new  cause.  The  limits  of  this 
obstacle  are  a  dependency  on  one  interpretation.  Analysts  may  place  too  much  importance  on 
what  they  know  about  a  person  or  situation  when  the  context  of  the  situation  may  be  the 
driving  factor.  The  problem  with  this  limit  is  the  analyst  will  misinterpret  the  evolving 
situation.  Analysts  tend  to  set  low  thresholds  for  acceptance  of  information.  They  confirm 
their  existing  assumptions  while  setting  high  thresholds  of  acceptance  when  the  information 
does  not  fit  their  assumptions.  The  downside  is  conclusions  tend  to  be  reinforced  arbitrarily 
instead  of  being  supported  by  data.  Finally,  analysts  tend  to  work  with  little  variation  in  or  no 
techniques  of  analysis.  Without  a  strategy,  the  analyst  cannot  effectively  attack  the  data. 
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Heuer’s  psychological  research  has  shown  three  types  of  cognitive  biases  exhibited  by 
intelligence  analysts:  evaluation  of  evidence,  perception  of  cause  and  effect,  and  estimation 

IS. 

of  probabilities.  First,  Heuer  identities  several  kinds  of  biases  in  the  evaluation  of 
evidence.  People  tend  to  believe  information  if  it  is  more  personal,  concrete  and  visual  than  if 
it  were  abstract.  The  negative  side  effect  of  this  bias  is  twofold:  analysts  give  more  weight  to 
anecdotal  and  personal  observations,  and  analysts  tend  to  distrust  statistical  data.  Analysts 
may  not  be  able  to  recognize  missing  evidence  they  need  to  confirm  a  situation,  nor  factor  this 
into  their  analysis.  As  a  result,  the  analyst  may  not  be  able  to  judge  the  impact  of  the  missing 
data  nor  accurately  estimate  their  confidence.  The  worst  case  is  the  analyst  may  not  know 
that  under  certain  circumstances  missing  data,  itself,  may  be  more  important  than  data  they 
possess.  Analysts  sometimes  overly  rely  on  consistency  as  a  check  for  evaluating  evidence. 
The  negative  effect  of  this  bias  is  the  analyst  may  place  too  much  emphasis  on  consistent  yet 
small  samples  of  data  resulting  in  false  conclusions.  Finally,  analysts  tend  to  encounter  data 
of  various  degrees  of  certainty  by  either  accepting  or  rejecting  it.  If  the  data  are  accepted, 
then  the  judgment  becomes  certain.  If  the  data  are  rejected,  the  judgment  becomes  certain. 

The  negative  effect  of  this  bias  is  uncertain  data  are  not  factored  into  the  analysis  because  of 
the  risk  of  failure  in  considering  a  lower  than  perfect  confidence. 

Second,  Heuer  identifies  a  number  of  biases  in  the  perception  of  cause  and  effect. 
Analysts  may  have  a  tendency  to  impose  order  onto  data  even  if  the  activity  is  random  or 
insufficient  data  exists  to  determine  causation.  The  negative  effect  of  this  bias  occurs  in 
complicated  situations  or  relationships  where  a  pattern  may  not  exist.  If  a  cause  is  speculated, 
it  may  very  well  be  false.  Analysts  have  a  tendency  to  interpret  causation  as  the  intentional 
result  of  a  centralization  tendency.  The  downside  of  this  bias  is  analysts  may  not  see  small 
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effects  or  unintended  consequences  leading  to  large  effects,  and,  therefore,  overestimate  the 
extent  logic  and  reason  play  in  a  situation.  This  bias  is  somewhat  characteristic  of  Chaos 
Theory.19  Analysts  tend  to  link  cause  and  effect  to  similar  relationships  (such  as  a  big  event 
must  have  a  big  result  or  cause).  The  negative  impact  of  this  bias  is  while  an  explanation 
sounds  more  logical  it  may  have  no  bearing  to  historical  precedence.  Analysts  tend  to 
overestimate  the  role  of  internal  factors  of  behavior  and  underestimate  the  role  of  external 
factors  (in  other  words,  not  enough  emphasis  is  placed  on  external  circumstances  surrounding 
an  individual  or  organization).  The  downside  of  this  bias  is  predictions  about  future  behavior 
of  an  individual  or  organizations  are  limited.  Analysts  tend  to  place  too  much  importance  on 
the  effect  of  their  own  importance  (for  example,  overestimating  the  impact  of  U.S.  policy) 
than  the  role  internal  processes  and  conflicts  play  within  other  individuals  and  organizations. 
The  negative  side  of  this  bias  is  analysts  miss  or  incorrectly  attribute  important  indicators 
driving  behavior  patterns.  Finally,  analysts  tend  to  conclude  because  they  can  correlate  two 
events  occurring  at  the  same  time  period,  one  causes  the  other.  The  downside  of  this  bias  is 
analysts  may  miss  the  fact  correlated  events  may  not  be  casually  related;  the  events  may  have 
a  common  cause  or  may  have  their  own  separate  causes  which  go  unnoticed. 

Last,  Heuer  identifies  biases  in  making  probability  estimates.  Analysts  tend  to 
qualitatively  judge  how  probable  an  event  is  by  recalling  similar  instances  from  memory.  The 
negative  impact  of  this  bias  is  twofold:  analysts  who  let  their  memory  be  the  judge  for 
statements  of  probability  may  miss  the  most  likely  probability  of  an  event,  and  analysts  must 
be  cognizant  consumers  of  intelligence  (policy-makers  and  decision-makers)  may  fall  victim 
to  the  same  bias.  Analysts  tend  to  be  overconfident  in  their  confidence  ranges.  They  premise 
their  estimate  on  previous  judgments  intuitively  or  unconsciously  set.  The  downside  of  this 
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bias  is  probabilities  may  be  somewhat  arbitrary,  especially  in  situations  with  incomplete 
information.  Analysts  tend  to  express  their  probabilities  of  certainty  using  words  like 
possible,  probable,  may  or  could.  A  more  rigorous  approach  is  to  use  quantitative  statements 
like  a  70  percent  probability,  assuming  quantification  is  possible.  The  negative  impact  of  this 
bias  is  verbal  estimates  can  lead  to  ambiguous  interpretations  leading  to  inaction  or  the  wrong 
action  by  policy-makers.  Finally,  analysts  tend  to  place  more  significance  with  specific  data 
about  an  individual  situation  than  historical  aggregate  data.  The  downside  of  this  bias  is 
analysts  may  not  leam  from  historical  data,  while  it  may  appear  to  have  no  causal  connection, 
it  actually  may  be  relevant. 

Heuer,  Clarkson  and  Schwartz,  a  leading  futurist,  suggest  how  to  compensate  for 
biases.  Schwartz’s  idea  is  to  generate  multiple  scenarios  for  the  future  to  “order”  analysts’ 
perceptions  of  the  present.  By  doing  so,  the  analyst  is  less  likely  to  fall  into  the  trap  of  only 
paying  attention  to  what  they  think  they  need  to  know."  Scenario  building  then  becomes  a 
self-reinforcing  learning  tool.  It  allows  the  analyst  to  be  more  aware  of  biases  and,  more 
importantly,  to  adapt  to  how  new  information  is  processed  by  the  analyst.  The  major 

challenges  to  building  scenarios  lie  with  the  analysts’  capability  to  be  intuitive  and 

22 

imaginative,  qualities  more  innate  than  trainable. 

Heuer  proposes  the  use  of  competing  hypotheses  to  help  evaluate  alternative 
explanations  and  minimize  some  of  the  analysts  cognitive  biases."  Heuer’ s  idea  is  based  on 
the  need  to  overcome  the  analyst’s  tendency  to  select  a  single  answer  they  intuitively  feel  is 
the  best.  Instead  of  picking  one  explanation,  analysts  pick  all  possible  explanations  within 
some  reasonable  parameters.  The  analyst  competes  them  against  each  other  instead  of 
evaluating  them  on  their  individual  merits.  One  important  characteristic  of  Heuer’ s  strategy  is 
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to  seek  evidence  to  refute  hypotheses,  somewhat  counter  to  most  analysts’  tendency  to  seek 
data  to  support  hypotheses.  Heuer,  like  Schwartz,  sees  his  approach  as  a  self-reinforcing 
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learning  tool  by  bringing  out  the  full  uncertainty  in  any  situation. 

Clarkson  envisions  the  day  when  intelligent  computers  can  capture  and  organize  data 
in  ways  the  human  mind  cannot.  By  so  doing,  computers  would  compensate  for  biases  that 
make  it  difficult  to  maintain  the  context  of  the  situation,  to  accurately  interpret  data,  and  to 
improve  the  analysts’  ability  to  imagine."  Clearly,  the  day  has  not  arrived  although  research 
into  agent-based  computer  modeling  may  be  promising  in  this  regard."  Clarkson  does, 
however,  point  out  a  very  important  consideration  -  motive.  Clarkson  believes  the 

intelligence  analyst  must  be  driven  by  the  motive  of  adventure,  which  the  analyst  seeks  out 

28 

novel  situations  and  relationships. 

Heuer,  Schwartz  and  Clarkson  provide  valuable  insights  into  the  limitations  of  mental 
processes.  The  identification  of  these  limitations  also  can  form  the  basis  -  through  research  - 
for  the  set  of  capabilities  agencies  would  want  their  future  analyst’s  to  possess.  What  is 
important  is  such  mental  capabilities  are  largely  inherent  within  the  individual.  They  are  not 
as  much  subject  to  development  through  training  and  education.  These  mental  capabilities 
that  overcome  biases  can  be  screened.  Schmidt  and  Hunter,  who  reviewed  85  years  of 
personnel  research  involving  thousands  of  studies  and  millions  of  employees,  concluded  the 
most  valid  predictor  for  entry-level  employees  was  cognitive  ability  tests."  However,  future 
research  is  needed  to  explicitly  translate  intelligence  analysis  bias-overcoming  capabilities 
identified  by  Heuer,  Schultz,  Clarkson  and  NSA  into  a  screening  program. 

The  recruitment  process  would  then  factor  these  capabilities  into  the  hiring  process. 
The  current  screening  process  for  entry-level  analysts  lies  largely  with  reviewing  resumes, 
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and  telephone  or  in-person  interviews  stating  what  one  has  done.  At  the  Defense  Intelligence 
Agency  (DIA),  the  main  entry-level  recruitment  processes  for  analysts  is  called  the  “candidate 
pool”  and  the  primary  analytical  screening  basis  involves  a  cover  letter  with  resume  and  a 
college  degree  may  not  be  required.  The  Central  Intelligence  Agency  (CIA)  is  essentially 
the  same  for  intelligence  analysts  although  the  security  screening  process  is  different.31  The 
National  Security  Agency  is  about  the  same  at  CIA. ,2  However,  there  are  many  examples  of 
professions  within  and  outside  of  government  with  a  systematic  screening  process. 

Types  of  Systematic  Screening  Processes 

There  are  two  basic  types  of  systematic  testing  used  in  pre-employment  screening.  In 
the  first  type,  a  systematic  screening  test  is  used  to  judge  success  for  follow-on  formal 
graduate  education  for  the  medical,  legal  and  business  professions,  for  example.  In  the 
second  type,  candidates  for  employment  must  take  systematic  screening  exams  for  over  100 
career  fields  in  the  federal  civil  service,  for  example,  but  there  is  no  follow-on  formal 
education  as  part  of  the  professional  screening.  Intelligence  analysts  take  neither  a 
screening  exam  nor  must  complete  formal  graduate  education  prior  to  employment. 

However,  there  are  examples  of  professions  using  mental  powers  similar  to  the  analyst  who 
do  require  screening  tests:  the  legal  profession  and  the  Department  of  State’s  Foreign  Service 
program. 

Comparing  the  entrance  requirements  for  the  legal  profession  and  the  Foreign  Service 
officer  to  the  all-source  analyst  is  not  entirely  coherent.  There  are  differences  in  the  career 
entrance  requirement,  in  the  case  of  law,  and  in  the  content  of  the  profession  and  different  use 
of  mental  skills  as  in  the  case  of  Foreign  Service.  However,  their  screening  exams  do  provide 
examples  of  how  professions  with  somewhat  comparable  intellectual  requirements  use  exams 
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although  they  are  used  differently.  The  law  exam  mostly  tests  mental  abilities  whereas  the 
Foreign  Service  exam  is  focused  more  on  interpersonal  relations  and  judgment. 

The  Law  School  Admissions  Test  (LSAT)  is  mainly  designed  to  see  if  the  candidate  is 
suitable  for  law  school.34  In  fact,  the  LSAT  does  not  test  for  knowledge  of  information, 
which  is  very  different  than  the  medical  college  admission  test.  The  LSAT  is  composed  of 
four  sections:  logical  reasoning,  logic  games,  reading  comprehension  and  writing  sample. 

The  logical  reasoning  section,  comprising  50  percent  of  the  exam,  is  designed  to  identify  the 
candidate’s  ability  to  understand,  analyze,  evaluate  and  manipulate  arguments  and  draw 

35 

reliable  conclusions  -  many  of  the  same  types  of  skills  used  by  the  intelligence  analyst. 

The  logic  games  section  tests  the  candidate’s  command  of  detail,  fonnal  deductive  abilities, 
how  rules  limit  and  order  behavior,  and  coping  with  many  pieces  of  data  simultaneously  to 

'Xfx 

solve  a  problem  -  many  of  these  abilities  are  used  by  the  intelligence  analyst.  The  reading 
comprehension  section  is  used  to  detennine  if  the  candidate  can  read  through  a  lot  of 
complicated  reading  material  -  skills  used  by  intelligence  analysts.  Finally,  the  writing 
sample  asks  the  candidate  to  argue  for  a  position  while  breaking  down  the  argument  of  the 

38 

opponent  -  somewhat  similar  skills  the  intelligence  analyst  must  face. 

The  Foreign  Service  exam  is  two-fold:  a  written  and  oral  exam.  The  candidate  must 
first  pass  an  eight  hour  written  exam.  It  is  made  up  of  three  sections,  although  just  the  first 
and  second  sections  are  substantive,  Job-Related  Knowledge  and  English  Expression, 
respectively.  The  knowledge  section  tests  what  the  candidates  knows  about  U.S.  history, 
government,  foreign  policy  and  foreign  political  systems.  The  English  section  tests  the 
candidate’s  knowledge  of  correct  grammar,  organization,  spelling  and  punctuation  as  well  as 
an  essay  section.  If  the  candidate  passes  the  written  exam,  an  oral  exam  is  scheduled. 
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The  Foreign  Service  oral  exam  can  last  eight  to  ten  hours.  The  assessment  measures 
the  following  skills  and  abilities:  written  communication,  oral  communication,  information 
integration  and  analysis,  planning  and  organizing,  judgment,  resourcefulness,  initiative  and 
leadership,  experience  and  motivation,  working  with  others,  composure,  quantitative  analysis, 
objectivity  and  integrity,  and  cultural  adaptability.40  The  oral  assessment  includes  a  group 
exercise,  a  structured  interview  and  a  management  exercise.  The  group  exercise  consists  of 
three  to  six  candidates  who  pretend  to  be  an  Embassy  staff  charged  with  making  decisions 
about  funding  competing  projects.  The  interview  focuses  on  the  candidate’s  judgment  and 
ability  to  think  on  their  feet.  The  management  exercise  tests  the  candidate’s  management 
skills,  interpersonal  skills  and  quantitative  ability. 

There  is  no  intent  to  suggest  the  LSAT  or  Foreign  Service  screening  processes  be  used 
in  part  or  in  whole  for  a  screening  program  for  the  intelligence  analyst.  They  do  have  content 
and  purpose  aspects,  which  could  be  used  as  a  starting  point  from  which  to  model  a  screening 
program  in  the  future.  The  content  of  an  analyst  screening  exam  might  look  more  like  the 
LSAT  but  populated  with  questions  based  on  researching  abilities  discussed  above  to 
compensate  for  mental  biases.  The  purpose  of  the  Foreign  Service  exam  is  more  comparable 
to  the  needs  of  an  analyst  screening  program.  There  is  no  educational  follow-on  step  required 
in  the  Foreign  Service  screening  process  like  there  is  for  the  legal,  medical  and  business 
professions,  which  suggests  the  relevance  of  the  Foreign  Service  case  to  analysts.  However,  a 
major  difference  is  the  capabilities  needed  for  analysis  are  more  enduring  than  the  Foreign 
Service,  as  the  following  case  study  suggests. 
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Evolution  of  the  Foreign  Service  Examination:  A  Case  Study 

From  the  signing  of  the  Constitution  in  1789  to  1856,  appointments  to  the  Diplomatic 
Service  -  representing  the  U.S.  government  in  overseas  Embassies  -  and  the  Consular  Service 
-  promoting  U.S.  commerce  -  were  made  politically,  not  on  merit.41  In  1856,  there  was  an 
attempt  to  modify  the  system  when  Congress  required  candidates  of  the  Consular  Service  to 
pass  an  examination,  however,  the  legislation  was  appealed  in  the  next  congressional 
session.  President  Grover  Cleveland  signed  an  Executive  Order  in  1895  requiring 
candidates  to  mid-level  positions  in  the  Consular  Service  to  pass  a  written  exam  on  consular 
regulations  and  an  oral  exam  demonstrating  proficiency  in  a  foreign  language.43  President 
Roosevelt  then  signed  Executive  Orders  in  1905  and  1906  to  require  lower-level  grades  in  the 
Consular  Service  and  for  secretaries  in  the  Diplomatic  Service  to  take  an  examination.44  The 
Rogers  Act  of  1924  merged  the  Consular  and  Diplomatic  Services  into  the  Foreign  Service  so 
all  candidates  into  the  Foreign  Service  would  take  the  written  and  oral  examination.  The  Act 
created  a  Board  of  Examiners  to  be  responsible  for  developing  the  rules  and  administration  of 
the  written  and  oral  examination.43 

Changes  to  the  examination  process  first  appeared  during  World  War  II.  Because 
there  were  not  enough  Foreign  Service  officers  at  the  beginning  of  World  War  II,  a  kind  of 
reserve  Foreign  Service  capability,  called  the  Auxiliary,  was  created.  Its  candidates  did  not 
take  the  written  and  oral  exams  given  during  the  nonnal  admissions  process.46  After  the  war, 
the  Foreign  Service  Organic  Act  of  1946  required  Presidential  appointees  to  the  Foreign 
Service  to  take  written  and  oral  examinations.47 

Post-World  War  II  reform  within  the  State  Department  included  the  need  for  entrance 
examination  changes  to  reflect  new  tasks  performed  by  Foreign  Service  officers.  By  1970, 
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one  major  government  study  cited  the  lack  of  creativity  in  its  Foreign  Service  officers.  The 
study  defined  creativity  as  “the  ability  to  fonnulate  new  combinations  from  two  or  more 
conceptions  already  in  mind.”49  The  study  considered  creativity  as  an  inborn  quality,  not  one 
taught.  Their  recommendation  was  to  update  the  written  exam  with  questions  to  test 
creativity  and  give  more  weight  to  those  questions.50  The  agency  contracted  to  produce  the 
exams,  the  Educational  Testing  Service,  said  it  would  be  possible  to  construct  exam  questions 
to  test  creativity,  but  the  technique  had  not  been  fully  validated  in  1970.M 

The  study  was  not  able  to  determine  if  creative  individuals  were  more  apt  to  take  the 
Foreign  Service  examinations,  but  they  were  able  to  determine  there  was  no  systematic 
process  to  attract  creative  people  into  taking  the  examinations.  "  Their  recommendation  was 
to  establish  a  network  of  people  with  foreign  service  experience  throughout  government, 

ST 

academia  and  business  who  could  help  identify  creative  candidates  to  take  the  examinations. 

The  study  found  the  oral  exam  -  no  longer  a  foreign  language  proficiency  test  -  the 
most  important  factor  in  the  admissions  process.54  However,  two  major  challenges  were 
cited:  the  volume  of  oral  assessments  overwhelmed  the  number  of  examiners;  and  the 
examiners  did  not  reflect  the  variety  of  experience  and  interests  within  the  Foreign  Service. 
The  study  recommended  not  every  candidate  who  passed  the  written  test  should  be  invited  to 
take  the  oral  exam,  and  oral  exam  panels  should  be  staffed  with  greater  diversity  within  the 
profession,  to  include  minorities  and  women.55 

The  study  also  cited  a  major  limitation  of  the  written  exam  corrected  just  prior  to 
1970;  the  exam  had  been  primarily  focused  on  academic  knowledge  -  no  longer  on 
regulations  -  but  did  not  test  a  candidate’s  aptitude  for  a  career  in  the  Foreign  Service.56  By 
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1970,  the  admissions  process  began  relying  more  on  the  oral  exam,  which  placed  more 
emphasis  on  aptitude.57 

The  Foreign  Service  Organic  Act  of  1980  was  passed  to  institutionalize  a  merit-based 
Foreign  Service.  '  As  a  result,  the  Act  updated  the  entrance  examination  and  made  them 
more  rigorous.  In  1946,  only  about  seven  percent  of  the  candidates  passed  both  written  and 
oral  exams;  by  2002,  the  proportion  had  dropped  to  about  two  percent  suggesting  the 
examination  process  had  indeed  been  made  more  rigorous.59 

High-level  government  commissions  reviewing  the  Foreign  Service  in  1989  reported 
the  examination  process  was  problematic  in  two  areas:  not  enough  women  and  minorities 
entered  the  service;  and  it  took  successful  candidates  too  long  to  receive  an  offer  of 
employment.60  The  commissions  recommended  the  then  standard  24-27  month  waiting 
period  from  the  time  the  candidate  took  the  written  exam  to  an  offer  of  employment  be 
reduced  to  six  months,  and  less  emphasis  be  placed  on  the  written  and  oral  examinations.61 

By  2002,  through  a  combination  of  the  after-effects  of  the  1 1  September  2001  terrorist 
attacks  and  the  leadership  qualities  of  Secretary  of  State  Powell,  the  number  of  applicants  to 
take  the  Foreign  Service  written  exam  in  fiscal  year  2002  doubled  to  23,000  from  the  amount 

in  fiscal  year  2001,  and  the  proportion  of  minorities  who  successfully  passed  the  written  exam 

62 

increased  to  17  percent,  the  highest  ever  percent  in  the  history  of  the  written  exam. 
Additionally,  the  time  lag  between  taking  the  written  exam  and  receiving  an  offer  of 

63 

employment  had  been  reduced  from  27  months  to  less  than  12  months. 

This  case  study  was  intended  to  be  representative  of  some  of  the  major  issues  faced  by 
the  Foreign  Service  screening  process.  Some  of  these  would  be  of  interest  to  the  development 
of  a  screening  program  for  intelligence  analysts.  These  include  but  are  not  limited  to  the  need 
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for  distinguishing  between  enduring  and  situational  capabilities,  the  need  for  and  challenge  of 
testing  for  creativity,  the  role  and  value  of  written  and  oral  assessments,  the  need  for  diversity, 
and  the  length  of  the  recruitment  time. 

If  a  screening  program  were  developed  for  intelligence  analysts,  then  introducing  a 
new  structure  into  the  recruitment  process  would  have  to  be  well  thought  out  and  there  would 
need  to  be  some  guidelines  for  measuring  its  success.  Organizational  theory  provides  a  sound 
theoretical  basis  for  implementing  a  systematic  screening  program.  Jones  separates  two  types 
of  organizational  changes:  evolutionary  and  revolutionary.64  Evolutionary  change  is  gradual 
and  incremental  while  revolutionary  change  is  sudden  and  drastic.65  Introducing  a  new 
screening  program  into  the  intelligence  community  has  to  be  evolutionary  because  there  is  no 
baseline  to  compare  or  measure  the  effect  of  envisioned  changes. 

Wallace  identifies  an  additional  characteristic  for  introducing  change  into  an 
organization:  having  a  plan.66  The  plan  can  range  from  structured  to  unstructured.  The 
reason  for  a  structured  change  is  to  support  a  solution  requiring  implementation  planning  and 
the  opportunity  to  modify  the  change  when  issues  come  up.  An  unstructured  change  is  best 
when  the  solution  is  not  clear.  For  the  screening  program,  a  structured  approach  would  be 
best  because  an  evolutionary  strategy  needs  an  implementation  plan. 

Senge  et  al  suggest  introducing  change  through  a  small  “incubation”  effort  because  the 
most  effective  type  of  change  is  one  where  employees  learn  to  change.  Additionally,  Senge 
states  change  requires  a  “self-perpetuating”  process  to  maintain  the  incentive  for  learning.69 
Authority-driven,  top-down  change  is  not  as  effective  according  to  Senge. 


17 


An  Intelligence  Analyst  Screening  Program:  Implementation  Plan 

Given  these  conceptual  frameworks,  a  three-phase  implementation  plan  spanning  three 
years  or  more  is  recommended.  The  first  phase,  composed  of  three  elements,  would  last  one 
year.  The  purpose  would  be  to  test  the  concept  using  current  employees  in  experimental  and 
control  groups.  The  screening  process  would  be  used  in  at  least  two  agencies,  such  as  the 
CIA  and  the  DIA.  In  the  first  element,  current  analysts  would  be  screened  to  create  a 
duplicate  analytical  effort,  such  as  an  aspect  of  counterterrorism  or  counterproliferation.  The 
experimental  group  would  be  composed  of  the  employees  who  “pass”  the  screening  process 
and  the  control  group  would  be  composed  of  an  existing  organization.  There  would  be  a 
combination  of  an  experimental  and  control  group  in  CIA  and  DIA,  each  focused  on  the  same 
analytical  area. 

The  second  element  of  the  first  phase  is  the  selection  of  a  supervisory  analyst  for  each 
experimental  group.  Davenport,  Prusak  and  Wilson  discuss  the  need  for  an  “ideas 
practitioner”  to  optimize  the  contributions  of  creative  employees.70  While  their  concept  is 
focused  on  the  business  community,  the  applicability  of  the  “ideas  practitioner”  to  other  kinds 
of  organizations  is  a  reasonable  assumption.  They  lay  out  the  kind  of  traits  and  managerial 
skills  needed  to  foster  innovation.  The  desired  traits  are  optimism,  a  passion  for  ideas  for 
their  own  sake,  and  the  ability  to  use  a  personal  network  beyond  the  chain-of-command  to  get 
help  when  needed.71  The  skills  are  what  distinguish  the  analyst  from  supervisor:  the  desire 
to  look  beyond  the  unit  for  new  approaches  to  solving  problems;  the  ability  to  connect  new 
ideas  to  the  organization’s  mission  and  customer;  the  ability  to  sell  ideas  to  senior  managers 
as  well  as  the  rank  and  file;  and  then  to  implement  a  project.  ~  Selection  of  this  individual 
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could  be  based  on  using  the  same  screening  program  as  the  analysts  in  addition  to  personal 
recommendations  from  senior  management. 

The  third  element  of  the  first  phase  is  the  selection  of  a  senior  leader  who  is  also  an 
idea-oriented  individual,  such  as  the  director  of  the  CIA  or  DIA.  This  leader  would  be  an 
advocate  for  the  experimental  group  and  its  supervisor  and  would  participate  in  the  review  of 
the  experimental  group’s  progress  and  support  decisions  about  the  group’s  recommendations. 
The  intent  is  not  for  the  senior  leader  to  be  biased  towards  the  experimental  group;  equal 
weight  is  given  to  both  the  results  of  experimental  and  control  groups. 

The  second  phase,  lasting  one  year,  would  build  upon  the  results  of  the  first  phase:  the 
use  of  the  screening  program  would  be  applied  on  a  very  limited  basis  as  a  pre-employment 
screening  tool  for  the  first  time  to  increase  the  size  of  the  experimental  groups  in  DIA  and 
CIA;  and  the  proof-of-concept  testing  used  in  phase  one  on  current  employees  at  DIA  and 
CIA  would  be  replicated  in  additional  organizations  like  the  Office  of  Naval  Intelligence 
(ONI)  and  the  National  Security  Agency  (NSA)  in  their  counterterrorism  or 
counterproliferation  units.  The  existing  employees  at  ONI  and  NSA  screened  would  include 
both  civilian  and,  for  the  first  time,  military.  The  control  groups  in  DIA  and  CIA  would  add 
the  same  number  of  staff  from  the  current  employee  pool  so  they  reach  parity  with  the 
experimental  groups. 

The  third  phase,  lasting  one  year,  would  build  upon  the  results  of  the  first  and  second 
phase:  the  use  of  the  screening  program  at  DIA  and  CIA  would  be  expanded  to  a  broader  pre¬ 
employment  analyst  candidate  pool  to  staff  additional  challenging  analytical  efforts;  ONI  and 
NSA  would  progress  to  phase  two  while  using  the  screening  program  on  both  new  civilians 
and  new  military  analysts;  and  the  proof-of-concept  testing  of  phase  one  would  be  replicated 
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into  other  parts  of  the  Intelligence  Community  such  as  the  Federal  Bureau  of  Investigation 
(FBI)  and  the  Department  of  Homeland  Security’s  (DHS)  Intelligence  Analysis  Center  (IAC). 
The  IAC  became  part  of  the  Intelligence  Community  with  the  passage  of  the  Homeland 
Security  Act  of  2002. 74 

Strategy  for  Measuring  the  Effectiveness  of  the  Implementation  Plan 

The  framework  for  measuring  effectiveness  lies  with  the  choice  of  a  quantitative  or 
qualitative  method.  Quantitative  methods  rely  on  scientifically  collected  data  and  analysis.  It 
is  not  clear  such  a  technique  is  applicable  for  comparing  the  activities  and  outputs  between  the 
experimental  and  control  groups.  A  qualitative  approach,  while  not  the  most  desirable,  is  the 
most  practical  and  achievable. 

Gibbs  discusses  two  types  of  qualitative  methods  for  consideration:  the  use  of  focus 
groups  and  the  use  of  observers.75  The  focus  group  would  consist  of  six  to  twelve  people  and 
include  analysts,  managers,  and  customers  from  the  analytical,  human  resources  and 
leadership  organizations.  The  focus  group  would  meet  monthly  during  each  phase  to  discuss 
their  opinions  about  the  activities  of  the  experimental  and  control  groups.  One  observer 
would  be  embedded  into  each  experimental  and  control  group  for  the  duration  of  each  phase 
and  would  record  activities  as  they  happen.  Their  report  would  be  provided  to  management 
on  a  recurring  basis. 

The  CIA’s  Shennan  Kent  School  for  Intelligence  Analysis  has  suggested  goals  that 
could  be  objectives  for  the  measurement  of  the  experimental  groups.  These  may  be 
appropriate  objectives  for  the  focus  groups  and  observers  to  use  as  measures  for  discussion 
and  observation,  respectively.  These  analytical  attributes  include:  innovation,  synthesis, 
learning,  questioning,  pattern  recognition,  adaptation  to  uncertainty,  visual  thinking, 
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experimentation,  metaphors,  nonlinear  systemic  thinking,  focus  on  what  is  unknown  or 
unknowable,  asking  “What  IF?,”  and  building  and  working  with  multiple  frameworks. 
Determining  how  these  objectives  would  be  recorded  for  measurement  and  comparison 
purposes  requires  additional  research. 

Throughout  the  phases,  the  focus  groups  and  observers  should  also  discuss  and  record 
the  effectiveness  of  communication  and  collaboration  within  the  experimental  groups  of  the 
various  agencies  and  separately  within  the  control  groups.  According  to  the  Kent  School, 
knowledge  sharing  will  be  one  of  the  most  critical  traits  needed  to  solve  complicated 
transnational  issues.77 

The  evaluation  process  can  also  be  used  to  address  the  issues  of  proportionality  and 
density.  How  much  of  an  organization  should  be  populated  by  systematically  screened 
employees?  For  example,  not  all  State  employees  take  the  Foreign  Service  exam.  Not  all 
functions  in  an  intelligence  analysis  organization  need  the  qualities  stated  throughout  the 
paper.  Can  fewer  screened  employees  outperform  larger  numbers  of  non-screened 
employees?  The  arguments  presented  throughout  this  paper  suggest  greater  efficiency  can  be 
achieved  with  fewer  employees. 

Finally,  measuring  the  effectiveness  of  the  systematic  screening  program  will  have  to 
include  observations  of  how  agencies  compensate  for  cultural  biases  and  power  relationships 
naturally  occurring  when  introducing  organizational  change. 

Conclusion 

This  paper  is  based  on  the  premise  we  can  identify  the  right  capabilities  for  optimizing 
analytical  work  and  that  we  should  bring  our  best  people  with  these  capabilities  together  to 
form  a  more  capable  Intelligence  Community.  Recommending  a  systematic  pre-employment 
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screening  program  for  intelligence  analysts  to  solve  this  shortfall  is  admittedly  a  bold  move. 
However,  aspects  of  this  recommendation  are  not  unprecedented.  CIA  and  NSA  have 
already  made  some  progress  in  defining  requirements.  Harvard  has  at  least  one  experimental 
research  program  useful  in  this  regard.79  However,  we  need  to  institutionalize  this  new 
paradigm  if  we  are  serious  about  overcoming  the  cognitive  and  conceptual  challenges  in 
analyzing  transnational  threats. 

Clearly  not  all  aspects  of  a  recommendation  were  solved  in  this  paper.  Some  were 
addressed  but  need  future  research:  the  nature  of  enduring  and  situational  mental  capabilities; 
the  proportion  of  employees  subject  to  a  systematic  screening  program;  the  content  and  form 
of  a  screening  program;  the  measurement  objectives  for  an  implementation  plan;  the  cultural 
resistance  to  introducing  a  screening  program;  and  validity  of  extrapolating  the  “ideas 
practitioner”  from  a  business  model  to  a  non-profit,  service  model.  There  was  no  attempt  to 
address  the  financial  costs  of  developing  such  a  programmatic  effort  prior  to  or  during 
implementation. 

It  will  take  a  few  years  to  develop  an  operational  screening  program.  Combined  with 
the  three  or  more  years  needed  for  implementation  testing  and  evaluation,  we  will  likely  not 
see  this  corporate  approach  operational  until  2010.  However,  if  we  do  not  start  the 
transformation  process  soon,  then  we  will  be  even  further  behind  in  adapting  to  the  post-Cold 
War  system.  It  is  a  system  characterized  by  the  increase  in  the  speed  of  communication, 
technological  and  normative  changes  and  the  complexity  of  emergent,  nonlinear, 
boundary  less  character  of  human  and  social  interactions. 
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